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TABLES  FOR  A  NEW  MULTIVARIATE 
GOODNESS-OF-FIT  TEST 


By 


R.  Franke  and  T.  Jayachandran 
Department  of  Mathematics 
Naval  Postgraduate  School 
Monterey,  CA  93940 


ABSTRACT 

We  present  tables  of  critical  values  for  a  new 
multivariate  goodness-of-f it  test  introduced  by 
Foutz.   Some  details  of  our  improved  asymptotic 
approximation  and  evaluation  of  its  accuracy 
are  given. 


1.   Introduction 

Foutz  [1]  proposed  a  new  goodness-of-f it  test  for  fitting 

univariate  as  well  as  multivariate  distributions.   He  showed 

that  the  null  distribution  of  the  test  statistic.  F  ,  does  not 

n 

depend  on  (1)  the  hypothesized  distribution,  or  (2)  the  number 
of  components  in  the  random  vector  under  study.   An  integral 

representation  for  the  null  CDF  of  F  was  provided.   Closed 

c  n 

form  expressions  for  this  null  distribution  are  quite 
difficult  to  obtain,  even  for  small  sample  sizes.   The 

alternative  has  been  to  approximate  the  distribution  by  a 

-1  -1     -2 

normal  distribution  with  mean  e   and  variance  (2e   -5e   )/n; 

this,  however,  does  not  appear  to  provide  a  good  approximation 

to  the  percentiles  of  the  null  distribution  of  F   for  moderate 

sample  sizes. 

The  authors [2]  compared  the  F  -test  with  the  Chi-squared 

test  and  the  Kolmogorov-Smirnov  test  and  found  that  the  F  -test 

n 

does  have  higher  power  when  fitting  certain  types  of  distribu- 
tions.  Another  investigation  by  the  authors  and  Linhart  [3] 
examined  the  power  of  the  F  -test  when  fitting  a  multivariate 
normal  distribution;  the  test  did  well  in  detecting  mean  shifts 

and  variance  shifts.   We  therefore  believe  that  the  F  -test 

n 

is  a  definite  alternative  to  the  Chi-squared  and  Kolmogorov- 
Smirnov  tests  when  fitting  univariate  distributions  and  it  is 
just  about  the  only  available  test  for  fitting  multivariate 
distributions.   However,  the  test  is  not  very  convenient  for 
applications  due  to  the  difficulty  in  obtaining  accurate 


critical  values.   This  paper  fills  the  gap  by  providing  tables 
of  approximate  pecentiles  of  the  null  distribution  of  F  . 

2.   Description  of  the  F  -Test 

r  n 

The  procedure  for  calculating  the  test  statistic  F   is  the 

following.   Given  a  random  sample  X,,  X  ,  ...  ,  X   ,,  from 

a  continuous  multivariate  distribution,  the  sample  space  is 

partitioned  into  n  statistically  equivalent  blocks.   Let 

h,(X),  h2 (X) ,  . . .  ,  h   , (X)  be  any  n-1  "cutting  functions" 

such  that  hn  (X)  has  a  continuous  distribution,  k  =  1. 
k  — 

2,  ...  ,  n-1,  and  let  k. ,  k  ,  ...  ,  k  _,  be  a  permutation  of 

1,  2,  ...  ,  n-1.   Let  X  (k, )  be  the  sample  vector  corresponding 

to  the  k, th  order  statistic  of  h   (X.),  i  =  1,  2,  ...  ,  n-1. 
i.  K-i   i 

The  initial  partition  of  the  sample  space  into  two  blocks  is 
defined  by 

B,  =  (X|h.   (X)  <  h7   (X(k_))},  and 

B2  =  B=. 
The  cutting  function  h,  (X)  is  then  used  to  partition  Bn 
(if  k2  <  kx)  or  B2  (if  k2  >  kx)  ^to  two  sunblocks  in  a  similar 
fashion.   When  all  the  cutting  functions  are  exhausted  the 
sample  space  will  have  been  partitioned  into  n  statistically 

equivalent  blocks,  $. ,    3~,  ...  ,  S  .   A  convenient  choice  for 

l    z  n 

the  cutting  functions  in  the  univariate  case  is  the  identity 
function.   In  the  multivariate  case  letting  h,  (X)  =  X  - '    ,    the 
jth  component  of  X  (for  various  j),  appears  to  work  well. 
More  details  on  partitioning  the  sample  space  into  statistically 
equivalent  blocks  and  some  examples  can  be  found  in  [3]. 


Once  the  statistically  equivalent  blocks  are  determined,  a 

computational  formula  for  the  test  statistic  F   for  the 

hypothesis  that  the  samples  are  from  a  specified  distribution 

H  is 

n  1 

Fn  =  E   max  [0,   ±   -  D±] , 

i=l 
where  D.  =  P[X  e  S . | H] 

The  integral  representation  for  the  null  CDF  of  F   results 
in  the  following  closed  form  expressions  for  n  =  3,  4,  and  5. 


P[F3  <  x]  =  " 
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It  does  not  appear  to  be  possible  to  generate  a  closed 

form  expression  for  the  CDF  of  F   in  the  general  case.   Foutz's 

n 

large  sample  normal  approximation  is  given  by 


(1) 


P[F   <  x]  =  <|> 
n 


n(x  -  e   ) 


,  ,.  -1    _  -2,  ,1/2 
(  (2e    -  5e   )  n) 


where  <J>  is  the  standard  normal  CDF.   To  check  the  accuracy  of 
this  approximation  in  our  earlier  study  [2],  we  generated 
samples  of  size  n-1  =  20,  30,  and  50  from  a  uniform  distribution 
on  [0,1]  and  tested  the  hypothesis  that  the  the  samples  are  in 
fact  from  that  distribution.   The  empirical  significance  levels 
in  80,000  replications  are  given  in  Table  1. 


Nominal 
Significance 

Level 

0.10 
0.05 
0.01 


n-1 


20 

0.0757 
0.0372 
0.0082 


30 

0.0800 
0.0399 
0.0083 


50 

0.0859 
0.0428 
0.0093 


Table  1 
Empirical  Significance  Level 
(Based  on  80^000  replications) 


It  can  be  seen  that  the  observed  significance  levels  are 
consistently  smaller  than  the  nominal  values  by  about  10-20%. 
We  therefore  proposed  the  use  of  Monte  Carlo  critical  values, 
which  were  based  on  25,000  replications.   These  values  are 
given  in  Table  2  and  the  corresponding  observed  significance 
levels,  base  -  on  225,000  subsequent  repetitions,  are  given  in 
Table  3. 


Significance 


n-1 


Level 20  30  50 


0.10  0.42714      0.41903      0.40816 

0.05  0.44865      0.43553      0.42116 

0.01  0.48659      0.46579      0.44487 


Table  2 
Monte  Carlo  Critical  Values 
(Based  on  25,000  replications) 


Nominal 
Significance 


n-1 


Level  20         30        50 


0.10  0.1006     0.9700    0.1003 

0.05  0.0486     0.0486    0.0498 

0.01  0.0103     0.0101    0.0102 


Table  3 
Empirical  Significance  Level 
(Based  on  225,000  replications) 


The  above  findings  lead  us  into  a  search  for  an  improved 
approximation  for  determining  the  percentiles  of  the  null 

distribution  of  F  .   We  found  that  allowing  the  mean  and  variance 

n  3 

to  be  functions  of  the  sample  size  leads  to  greatly  improved 
approximations.   While  it  is  difficult  to  give  precise  error 
bounds  on  the  percentile  values,  our  computational  experience 
indicates  about  a  four  decimal  place  accuracy.   This  leads  to 
rejection  rates  with  errors  in  the  fourth  decimal  place,  usually. 
Comparing  the  error  in  the  rejection  rates  for  the  asymptotic 
approximation  (1)  given  by  Foutz,  our  approximation  is  better 
by  a  factor  of  10  or  more. 


3.   Modified  Normal  Approximation 

The  data  for  the  approximation  of  the  null  distribution  of 
the  Foutz  statistic  was  obtained  by  Monte  Carlo  methods.   For 
a  given  sample  size  n-1,  sequences  of  n-1  uniformly  distributed 
numbers  where  generated  using  the  IMSL*  random  number  generator 
GGUBS.   The  Foutz  statistic  was  then  computed  and  tabulated 
into  one  of  200  equilength  intervals.   This  process  was 
replicated  25,000  times.   The  entire  set  consists  of  the 
empirical  cumulative  distribution  functions  obtained  from  this 
data  for  60  sample  sizes,  n-1  =  2(1)40,  40(2)70,  and  70(5)100. 
Potentially  this  yields  as  many  as  12000  pieces  of  data, 
however  if  only  intervals  with  nontrivial  data  in  them  are 
counted,  this  is  reduced  to  about  4700. 

A  data  fitting  problem  with  4700  points  is  not  easily  handled 
unless  a  linear  model  is  accepted.   We  do  not  know  the  behavior 
of  the  distribution  as  the  sample  size  gets  large,  so  we  were 
reluctant  to  impose  a  form  with  only  linear  parameters, 
especially  in  sample  size.   We  decided  on  attempting  a  correction 
to  the  asymptotic  approximation  given  by  Foutz. 

After  some  experimentation  with  various  types  of  corrections, 
it  was  decided  the  most  reasonable  was  to  include  correction 
terms  in  the  argument  of  the  asymptotic  approximation.   In 
order  to  make  the  computation  feasible  it  was  decided  to  fit  the  data 
in  a  two  pass  scheme;  first  the  null  distribution  for  each  sample 
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size  was  approximated  as  below,  and  then  the  parameters  in  these 
approximations  were  fit  by  functions  of  sample  size. 

The  precise  form  of  the  approximation  was  through  the 
argument  of  a  normal  distribution,  which  was  taken  to  be  of 
the  form 

(a  +  b  nCx-e"1)  +  c  (x-e"1)2)/  >/(2  _i  -  5e~2)n. 

Because  we  are  strongly  interested  in  the  inverse  CDF,  the  data 
was  weighted  at  each  point  by  the  centered  difference  from  the 
Monte  Carlo  data,  which  then  resulted  in  a  greater  weight  on 
the  part  of  the  curve  with  a  large  slope.   The  results  of  this 
least  squares  process  yielded  a  table  of  values  of  a,  b,  and 
c  versus  sample  size  (actually  we  consider  them  as  functions 
of  n  =  sample  size  +  1)  .   We  observe  that  the  amount  of 
scatter  increases  as  n  increases.   There  tends  to  be  even  more 
scatter  with  higher  powers  of  (x-e   ) .   For  this  reason  it  was 
decided  to  weight  the  smaller  sample  sizes  more  heavily,  and  a 
weight  of  1/n  was  adopted.   Since  the  data  is  more  dense  for 
smaller  sample  sizes  this  results  in  considerably  less  weight 
for  the  large  sample  sizes,  although  we  feel  the  trend  is  still 
properly  modelled  and  that  our  approximation  is  considerably 
better  than  the  asymptotic  approximation  for  very  large  sample 
sizes,  say  even  up  to  1000. 

In  the  second  stage  of  the  process  the  coefficients  a,  b, 
and  c  was  chosen  to  allow  a  rate  of  decay  (or  growth)  of  the 
coefficients  to  be  dictated  by  the  data.   Thus  we  fit  a,  b  and 
c  with  functions  of  the  form  A  +  Bn  . 


For  the  terms  which  are  constant  and  linear  in  (x-e   )  the 
exponent  was  negative,  however,  for  C (n)  the  exponent  was 
positive,  indicating  that  the  term  grows  (somewhat  slower  than 
linearly)  with  sample  size.   We  do  not  consider  this  as 
bothersome,  however,  since  the  linear  term  in  (x-e   ) 
has  already  (due  to  the  form  of  the  asymptotic  approximation) 
been  included  with  a  factor  that  grows  linearly  with  sample 
size. 

The  overall  result  of  this  nonlinear  least  squares 
approximation  is  the  approximate  CDF  involving  the  nine 
parameters , 


(2)  P[Fn    <   x]       <j>    [(g(x>/    V(2e    l   -    5    e~2)n)l     , 

-1  -1    2 

where   g(x)    =    a(n)    +   b(n)    n    (x-e      )    +   c (n)     (x-e      )     ,    and 

a(n)  =  0.2039  +  0.1876  n"1-4416' 
b(n)  =  1.0015  -  0.05672  n"0*7377, 
c(n)  =  0.3049  -  0.5912  n0,8927. 

In  order  to  test  our  results,  two  different  approaches 
were  taken.   First,  the  number  of  rejections  for  previously  run 
tests  were  available  for  sample  sizes  of  n-1  =20,  30,  and  50, 
at  (approximately)  the  0.10,  0.05,  and  0.01  levels.   3y 
computing  the  derivative  of  the  approximate  CDF,  equation  (2), 
and  making  a  correction  along  the  tangent  line,  we  were  able  to 
estimate  the  anticipated  rejection  rate  that  would  occur  with 
our  present  approximation.   This  data  was  accumulated  over 
225,000  replications,  and  is  given  in  Table  4.   The  main  entry 


is  the  anticipated  rejection  rate  when  using  the  results  of  our 
approximation,  above.   As  a  point  of  comparision  with  Foutz * s 
asymptotic  approximation,  we  include  the  corresponding  rates 
for  it  in  parenthesis.   Second,  to  test  the  approximation  for 
a  smaller,  as  well  as  an  intermediate  sample  size,  we  computed 
the   Foutz  statistic  for  300,000  uniformly  distributed  samples 
of  sizes  10  and  40,  and  tabulated  them  at  intervals  of  .0001  in 
the  range  of  interest.   The  results  of  these  calculations  are 
shown  in  Table  5  for  the  0.10,  0.05,  and  0.01  levels. 


Nominal 
Significance 


n-1 


Level  20  30        50 


0.10  0.0994      0.1002     0.1007 

(0.0764)    (0.0801)   (0.0840) 

0.05  0.0496      0.0500     0.0505 

(0.0385)    (0.0402)   (0.0420) 

0.01  0.0098      0.0095     0.0098 

(0.0085)    (0.0086)   (0.0088) 

Table  4 
Anticipated  Rejection  Rates 
From  Approximate  Critical  Values 
(Based  on  225,000  replications) 


Nominal  n_]_ 

Significance 


Level  10  40 


0.10                        0.0989  0.0998 

(0.0687)  (0.0824) 

0.05                        0.0481  0.0491 

(0.0349)  (0.0087) 

0.01                        0.0086  0.0098 

(0.0069)  (0.0087) 

Table  5 
Empirical  Significance  Levels 
(Base  on  300,000  replications) 


As  is  shown  by  the  tables,  we  expect  the  error  in  the 
rejection  rates  due  to  use  of  our  approximate  percentiles  to 
be  smaller  by  a  factor  of  10-20  for  the  0.20  to  0.05  level 
than  they  are  for  Foutz ' s  normal  approximation.   At  the 
extreme  tails,  our  approximation  is  not  as  good  as  at  the  more 
moderate  levels,  but  is  still  a  worthwhile  improvement  over  the 
asymptotic  approximation. 

Table  6  lists  some  upper  percentiles  of  the  approximate 
CDF  given  by  Equation  (2)  for  sample  sizes  5(1)100,  100(10)200, 
and  200(100)1000.   The  exact  values  are  given  for  n-1  =  2,  3, 
and  4.   Since  we  expect  the  entries  to  have  about  4  digit 
accuracy,  linear  interpolation  for  intermediate  sample  sizes 
will  have  comparable  accuracy.   Linear  interpolation  in  the 
percentiles  is  not  accurate,  and  other  percentiles  should  be 
calculated  from  equation  (.2)  .   It  is  interesting  to  observe  the 
"surface"  of  the  null  CDF  in  a  perspective  plot,  as  in  Figure 
1.   Of  course,  only  discrete  slices  exist;  the  cross  section 
lines  in  the  direction  of  sample  size  are  an  artifact  of  the 
plotting  package.   The  convergence  toward  a  sharp  rise  of  the 
CDF  in  the  vicinity  of  x  -  e    as  sample  size  increases  is  very 
apparent. 
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TABLE     6 
Approximate    percentage    poinds    for    the   null 
distribution    of   the    Foutz    statistic 
(Note:      sample    size   is    n-1) 


*      from   exact    distribution 


n-1         0.80  0.85  0.90  3.95  0.975  0.99  0.995 
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